The Chatbot Usability Scale: the Design and Pilot of a Usability Scale for Interaction with AI-Based Conversational Agents

Abstract Standardised tools to assess a user’s satisfaction with the experience of using chatbots and conversational agents are currently unavailable. This work describes four studies, including systematic literature review, an overall sample 141 participants in survey (experts novices), focus group sessions testing (i) define attributes quality interaction (ii) designing piloting new scale measure after chatbots. Two instruments were developed: A diagnostic tool form checklist (BOT-Check). is development previous works which can be used reliably check line commonplace principles. 15-item questionnaire (BOT Usability Scale, BUS-15) estimated reliability between .76 .87 distributed five factors. BUS-15 strongly correlates UMUX-LITE by enabling designers consider broader range aspects usually not considered for non-conversational agents, e.g. efficiency accessibility, chatbot’s functionality so on. Despite convincing psychometric properties, requires further validation. Designers use it as products, thus building independent databases future evaluation its reliability, validity sensitivity.